

<!DOCTYPE html>
<html class="writer-html5" lang="zh" >
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Scrapy 教程 &mdash; Scrapy 2.3.0 文档</title>
  

  
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.custom.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster.bundle.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-shadow.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-punk.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-noir.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-light.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/tooltipster-sideTip-borderless.min.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/micromodal.css" type="text/css" />
  <link rel="stylesheet" href="../_static/css/sphinx_rtd_theme.css" type="text/css" />

  
  
  
  

  
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/language_data.js"></script>
        <script src="../_static/js/hoverxref.js"></script>
        <script src="../_static/js/tooltipster.bundle.min.js"></script>
        <script src="../_static/js/micromodal.min.js"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    
    <link rel="index" title="索引" href="../genindex.html" />
    <link rel="search" title="搜索" href="../search.html" />
    <link rel="next" title="实例" href="examples.html" />
    <link rel="prev" title="安装指南" href="install.html" /> 

<script data-ad-client="ca-pub-1466963416408457" async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>


<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="../index.html" class="icon icon-home" alt="Documentation Home"> Scrapy
          

          
          </a>

          
            
            
              <div class="version">
                2.3
              </div>
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <p class="caption"><span class="caption-text">第一步</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="overview.html">Scrapy一目了然</a></li>
<li class="toctree-l1"><a class="reference internal" href="install.html">安装指南</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Scrapy 教程</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#creating-a-project">创建项目</a></li>
<li class="toctree-l2"><a class="reference internal" href="#our-first-spider">我们的第一只蜘蛛</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#how-to-run-our-spider">如何运行我们的蜘蛛</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#what-just-happened-under-the-hood">引擎盖下面发生了什么？</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#a-shortcut-to-the-start-requests-method">启动请求方法的快捷方式</a></li>
<li class="toctree-l3"><a class="reference internal" href="#extracting-data">提取数据</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#xpath-a-brief-intro">XPath: 简介</a></li>
<li class="toctree-l4"><a class="reference internal" href="#extracting-quotes-and-authors">提取引用和作者</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#extracting-data-in-our-spider">在蜘蛛中提取数据</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#storing-the-scraped-data">存储抓取的数据</a></li>
<li class="toctree-l2"><a class="reference internal" href="#following-links">以下链接</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#a-shortcut-for-creating-requests">创建请求的快捷方式</a></li>
<li class="toctree-l3"><a class="reference internal" href="#more-examples-and-patterns">更多示例和模式</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#using-spider-arguments">使用蜘蛛参数</a></li>
<li class="toctree-l2"><a class="reference internal" href="#next-steps">下一步</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="examples.html">实例</a></li>
</ul>
<p class="caption"><span class="caption-text">基本概念</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../topics/commands.html">命令行工具</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/spiders.html">蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/selectors.html">选择器</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/items.html">项目</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/loaders.html">项目加载器</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/shell.html">Scrapy shell</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/item-pipeline.html">项目管道</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/feed-exports.html">Feed 导出</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/request-response.html">请求和响应</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/link-extractors.html">链接提取器</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/settings.html">设置</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/exceptions.html">例外情况</a></li>
</ul>
<p class="caption"><span class="caption-text">内置服务</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../topics/logging.html">登录</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/stats.html">统计数据集合</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/email.html">发送电子邮件</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/telnetconsole.html">远程登录控制台</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/webservice.html">Web服务</a></li>
</ul>
<p class="caption"><span class="caption-text">解决具体问题</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">常见问题</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/debug.html">调试spiders</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/contracts.html">蜘蛛合约</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/practices.html">常用做法</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/broad-crawls.html">宽爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/developer-tools.html">使用浏览器的开发人员工具进行抓取</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/dynamic-content.html">选择动态加载的内容</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/leaks.html">调试内存泄漏</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/media-pipeline.html">下载和处理文件和图像</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/deploy.html">部署蜘蛛</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/autothrottle.html">AutoThrottle 扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/benchmarking.html">标杆管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/jobs.html">作业：暂停和恢复爬行</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/coroutines.html">协同程序</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/asyncio.html">asyncio</a></li>
</ul>
<p class="caption"><span class="caption-text">扩展Scrapy</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../topics/architecture.html">体系结构概述</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/downloader-middleware.html">下载器中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/spider-middleware.html">蜘蛛中间件</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/extensions.html">扩展</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/api.html">核心API</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/signals.html">信号</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/exporters.html">条目导出器</a></li>
</ul>
<p class="caption"><span class="caption-text">其余所有</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../news.html">发行说明</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">为 Scrapy 贡献</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versioning.html">版本控制和API稳定性</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Scrapy</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
        
      <li>Scrapy 教程</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="scrapy-tutorial">
<span id="intro-tutorial"></span><h1>Scrapy 教程<a class="headerlink" href="#scrapy-tutorial" title="永久链接至标题">¶</a></h1>
<p>在本教程中，我们假定scrapy已经安装在您的系统上。如果不是这样的话，看 <a class="reference internal" href="install.html#intro-install"><span class="std std-ref">安装指南</span></a> .</p>
<p>我们将抓取' quotes.toscrape.com &lt;http: quotes.toscrape.com=&quot;&quot;&gt;&lt;/http:&gt; ' _，这是一个列出著名作家名言的网站。</p>
<p>本教程将指导您完成以下任务：</p>
<ol class="arabic simple">
<li><p>创建新的Scrapy项目</p></li>
<li><p>写一篇 <a class="reference internal" href="../topics/spiders.html#topics-spiders"><span class="std std-ref">spider</span></a> 对网站进行爬网并提取数据</p></li>
<li><p>使用命令行导出抓取的数据</p></li>
<li><p>将spider改为递归跟踪链接</p></li>
<li><p>使用蜘蛛参数</p></li>
</ol>
<p>Scrapy是用 Python 写的。如果你对这门语言不熟悉，你可能想从了解这门语言是什么开始，从 Scrapy 语言中得到最大的收获。</p>
<p>如果您已经熟悉其他语言，并且希望快速学习Python，那么 <a class="reference external" href="https://docs.python.org/3/tutorial">Python Tutorial</a> 是一种很好的资源。</p>
<p>如果您是编程新手，并且想从python开始，那么下面的书可能对您有用：</p>
<ul class="simple">
<li><p><a class="reference external" href="https://automatetheboringstuff.com/">Automate the Boring Stuff With Python</a></p></li>
<li><p><a class="reference external" href="http://openbookproject.net/thinkcs/python/english3e/">How To Think Like a Computer Scientist</a></p></li>
<li><p><a class="reference external" href="https://learnpythonthehardway.org/python3/">Learn Python 3 The Hard Way</a></p></li>
</ul>
<p>你也可以看看 <a class="reference external" href="https://wiki.python.org/moin/BeginnersGuide/NonProgrammers">this list of Python resources for non-programmers</a> 以及 <a class="reference external" href="https://www.reddit.com/r/learnpython/wiki/index#wiki_new_to_python.3F">suggested resources in the learnpython-subreddit</a> .</p>
<div class="section" id="creating-a-project">
<h2>创建项目<a class="headerlink" href="#creating-a-project" title="永久链接至标题">¶</a></h2>
<p>在开始抓取之前，你必须建立一个新的零碎项目。输入要在其中存储代码并运行的目录：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">startproject</span> <span class="n">tutorial</span>
</pre></div>
</div>
<p>这将创建一个 <code class="docutils literal notranslate"><span class="pre">tutorial</span></code> 目录包含以下内容：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">tutorial</span><span class="o">/</span>
    <span class="n">scrapy</span><span class="o">.</span><span class="n">cfg</span>            <span class="c1"># deploy configuration file</span>

    <span class="n">tutorial</span><span class="o">/</span>             <span class="c1"># project&#39;s Python module, you&#39;ll import your code from here</span>
        <span class="fm">__init__</span><span class="o">.</span><span class="n">py</span>

        <span class="n">items</span><span class="o">.</span><span class="n">py</span>          <span class="c1"># project items definition file</span>

        <span class="n">middlewares</span><span class="o">.</span><span class="n">py</span>    <span class="c1"># project middlewares file</span>

        <span class="n">pipelines</span><span class="o">.</span><span class="n">py</span>      <span class="c1"># project pipelines file</span>

        <span class="n">settings</span><span class="o">.</span><span class="n">py</span>       <span class="c1"># project settings file</span>

        <span class="n">spiders</span><span class="o">/</span>          <span class="c1"># a directory where you&#39;ll later put your spiders</span>
            <span class="fm">__init__</span><span class="o">.</span><span class="n">py</span>
</pre></div>
</div>
</div>
<div class="section" id="our-first-spider">
<h2>我们的第一只蜘蛛<a class="headerlink" href="#our-first-spider" title="永久链接至标题">¶</a></h2>
<p>蜘蛛是你定义的类，Scrapy用来从一个网站（或一组网站）获取信息。它们必须是子类 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider" title="scrapy.spiders.Spider"><code class="xref py py-class docutils literal notranslate"><span class="pre">Spider</span></code></a> 定义要发出的初始请求，可以选择如何跟踪页面中的链接，以及如何解析下载的页面内容以提取数据。</p>
<p>这是我们第一只蜘蛛的代码。将其保存在名为的文件中 <code class="docutils literal notranslate"><span class="pre">quotes_spider.py</span></code> 下 <code class="docutils literal notranslate"><span class="pre">tutorial/spiders</span></code> 项目中的目录：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">QuotesSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;quotes&quot;</span>

    <span class="k">def</span> <span class="nf">start_requests</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">urls</span> <span class="o">=</span> <span class="p">[</span>
            <span class="s1">&#39;http://quotes.toscrape.com/page/1/&#39;</span><span class="p">,</span>
            <span class="s1">&#39;http://quotes.toscrape.com/page/2/&#39;</span><span class="p">,</span>
        <span class="p">]</span>
        <span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">urls</span><span class="p">:</span>
            <span class="k">yield</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="o">=</span><span class="n">url</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="n">page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">url</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;/&quot;</span><span class="p">)[</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
        <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;quotes-</span><span class="si">{</span><span class="n">page</span><span class="si">}</span><span class="s1">.html&#39;</span>
        <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;wb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
            <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Saved file </span><span class="si">{</span><span class="n">filename</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
</pre></div>
</div>
<p>如你所见，我们的蜘蛛子类 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider" title="scrapy.spiders.Spider"><code class="xref py py-class docutils literal notranslate"><span class="pre">scrapy.Spider</span></code></a> 并定义了一些属性和方法：</p>
<ul>
<li><p><a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.name" title="scrapy.spiders.Spider.name"><code class="xref py py-attr docutils literal notranslate"><span class="pre">name</span></code></a> ：标识蜘蛛。它在一个项目中必须是唯一的，也就是说，不能为不同的蜘蛛设置相同的名称。</p></li>
<li><p><a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.start_requests" title="scrapy.spiders.Spider.start_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start_requests()</span></code></a> ：必须返回一个ITable of requests（您可以返回一个请求列表或编写一个生成器函数），蜘蛛将从中开始爬行。随后的请求将从这些初始请求中依次生成。</p></li>
<li><p><a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.parse" title="scrapy.spiders.Spider.parse"><code class="xref py py-meth docutils literal notranslate"><span class="pre">parse()</span></code></a> ：将调用的方法，用于处理为每个请求下载的响应。响应参数是的实例 <a class="reference internal" href="../topics/request-response.html#scrapy.http.TextResponse" title="scrapy.http.TextResponse"><code class="xref py py-class docutils literal notranslate"><span class="pre">TextResponse</span></code></a> 它保存页面内容，并有进一步有用的方法来处理它。</p>
<p>这个 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.parse" title="scrapy.spiders.Spider.parse"><code class="xref py py-meth docutils literal notranslate"><span class="pre">parse()</span></code></a> 方法通常解析响应，将抓取的数据提取为dict，并查找新的URL以跟踪和创建新的请求。（ <a class="reference internal" href="../topics/request-response.html#scrapy.http.Request" title="scrapy.http.Request"><code class="xref py py-class docutils literal notranslate"><span class="pre">Request</span></code></a> 从他们那里。</p>
</li>
</ul>
<div class="section" id="how-to-run-our-spider">
<h3>如何运行我们的蜘蛛<a class="headerlink" href="#how-to-run-our-spider" title="永久链接至标题">¶</a></h3>
<p>要使蜘蛛正常工作，请转到项目的顶级目录并运行：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">crawl</span> <span class="n">quotes</span>
</pre></div>
</div>
<p>此命令运行名为的spider <code class="docutils literal notranslate"><span class="pre">quotes</span></code> 我们刚刚添加的，这将发送一些 <code class="docutils literal notranslate"><span class="pre">quotes.toscrape.com</span></code> 领域。您将得到类似于以下内容的输出：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">...</span> <span class="p">(</span><span class="n">omitted</span> <span class="k">for</span> <span class="n">brevity</span><span class="p">)</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Spider</span> <span class="n">opened</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">extensions</span><span class="o">.</span><span class="n">logstats</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Crawled</span> <span class="mi">0</span> <span class="n">pages</span> <span class="p">(</span><span class="n">at</span> <span class="mi">0</span> <span class="n">pages</span><span class="o">/</span><span class="nb">min</span><span class="p">),</span> <span class="n">scraped</span> <span class="mi">0</span> <span class="n">items</span> <span class="p">(</span><span class="n">at</span> <span class="mi">0</span> <span class="n">items</span><span class="o">/</span><span class="nb">min</span><span class="p">)</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">extensions</span><span class="o">.</span><span class="n">telnet</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Telnet</span> <span class="n">console</span> <span class="n">listening</span> <span class="n">on</span> <span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="p">:</span><span class="mi">6023</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Crawled</span> <span class="p">(</span><span class="mi">404</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">robots</span><span class="o">.</span><span class="n">txt</span><span class="o">&gt;</span> <span class="p">(</span><span class="n">referer</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Crawled</span> <span class="p">(</span><span class="mi">200</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">1</span><span class="o">/&gt;</span> <span class="p">(</span><span class="n">referer</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Crawled</span> <span class="p">(</span><span class="mi">200</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">2</span><span class="o">/&gt;</span> <span class="p">(</span><span class="n">referer</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">quotes</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Saved</span> <span class="n">file</span> <span class="n">quotes</span><span class="o">-</span><span class="mf">1.</span><span class="n">html</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">quotes</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Saved</span> <span class="n">file</span> <span class="n">quotes</span><span class="o">-</span><span class="mf">2.</span><span class="n">html</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">16</span> <span class="mi">21</span><span class="p">:</span><span class="mi">24</span><span class="p">:</span><span class="mi">05</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">INFO</span><span class="p">:</span> <span class="n">Closing</span> <span class="n">spider</span> <span class="p">(</span><span class="n">finished</span><span class="p">)</span>
<span class="o">...</span>
</pre></div>
</div>
<p>现在，检查当前目录中的文件。您应该注意到已经创建了两个新文件： <em>quotes-1.html</em> 和 引用-2.HTML, 将各个URL的内容作为 <code class="docutils literal notranslate"><span class="pre">parse</span></code> 方法指示。</p>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>如果您想知道为什么我们还没有解析HTML，请稍等，我们很快就会讨论这个问题。</p>
</div>
<div class="section" id="what-just-happened-under-the-hood">
<h4>引擎盖下面发生了什么？<a class="headerlink" href="#what-just-happened-under-the-hood" title="永久链接至标题">¶</a></h4>
<p>Scrapy安排了 <a class="reference internal" href="../topics/request-response.html#scrapy.http.Request" title="scrapy.http.Request"><code class="xref py py-class docutils literal notranslate"><span class="pre">scrapy.Request</span></code></a> 返回的对象 <code class="docutils literal notranslate"><span class="pre">start_requests</span></code> 蜘蛛的方法。在接收到每个响应时，它实例化 <a class="reference internal" href="../topics/request-response.html#scrapy.http.Response" title="scrapy.http.Response"><code class="xref py py-class docutils literal notranslate"><span class="pre">Response</span></code></a> 对象并调用与请求关联的回调方法（在本例中，为 <code class="docutils literal notranslate"><span class="pre">parse</span></code> 方法）将响应作为参数传递。</p>
</div>
</div>
<div class="section" id="a-shortcut-to-the-start-requests-method">
<h3>启动请求方法的快捷方式<a class="headerlink" href="#a-shortcut-to-the-start-requests-method" title="永久链接至标题">¶</a></h3>
<p>而不是执行 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.start_requests" title="scrapy.spiders.Spider.start_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start_requests()</span></code></a> 生成的方法 <a class="reference internal" href="../topics/request-response.html#scrapy.http.Request" title="scrapy.http.Request"><code class="xref py py-class docutils literal notranslate"><span class="pre">scrapy.Request</span></code></a> 来自URL的对象，您只需定义 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.start_urls" title="scrapy.spiders.Spider.start_urls"><code class="xref py py-attr docutils literal notranslate"><span class="pre">start_urls</span></code></a> 具有URL列表的类属性。然后，此列表将由 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.start_requests" title="scrapy.spiders.Spider.start_requests"><code class="xref py py-meth docutils literal notranslate"><span class="pre">start_requests()</span></code></a> 要为您的蜘蛛创建初始请求，请执行以下操作：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">QuotesSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;quotes&quot;</span>
    <span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s1">&#39;http://quotes.toscrape.com/page/1/&#39;</span><span class="p">,</span>
        <span class="s1">&#39;http://quotes.toscrape.com/page/2/&#39;</span><span class="p">,</span>
    <span class="p">]</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="n">page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">url</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;/&quot;</span><span class="p">)[</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
        <span class="n">filename</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">&#39;quotes-</span><span class="si">{</span><span class="n">page</span><span class="si">}</span><span class="s1">.html&#39;</span>
        <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">&#39;wb&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
            <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
</pre></div>
</div>
<p>这个 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.parse" title="scrapy.spiders.Spider.parse"><code class="xref py py-meth docutils literal notranslate"><span class="pre">parse()</span></code></a> 方法将被调用来处理这些URL的每个请求，即使我们没有明确地告诉Scrapy这样做。这是因为 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.Spider.parse" title="scrapy.spiders.Spider.parse"><code class="xref py py-meth docutils literal notranslate"><span class="pre">parse()</span></code></a> 是Scrapy的默认回调方法，对没有显式分配回调的请求调用该方法。</p>
</div>
<div class="section" id="extracting-data">
<h3>提取数据<a class="headerlink" href="#extracting-data" title="永久链接至标题">¶</a></h3>
<p>学习如何使用scrappy提取数据的最佳方法是使用 <a class="reference internal" href="../topics/shell.html#topics-shell"><span class="std std-ref">Scrapy shell</span></a> . 运行：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">shell</span> <span class="s1">&#39;http://quotes.toscrape.com/page/1/&#39;</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">注解</p>
<p>否则，在运行Scrapy命令时，请记住要在命令行中包含url。 <code class="docutils literal notranslate"><span class="pre">&amp;</span></code> 字符）不起作用。</p>
<p>在Windows上，使用双引号：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">shell</span> <span class="s2">&quot;http://quotes.toscrape.com/page/1/&quot;</span>
</pre></div>
</div>
</div>
<p>您将看到类似的内容：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">[</span> <span class="o">...</span> <span class="n">Scrapy</span> <span class="n">log</span> <span class="n">here</span> <span class="o">...</span> <span class="p">]</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">19</span> <span class="mi">12</span><span class="p">:</span><span class="mi">09</span><span class="p">:</span><span class="mi">27</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">engine</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Crawled</span> <span class="p">(</span><span class="mi">200</span><span class="p">)</span> <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">1</span><span class="o">/&gt;</span> <span class="p">(</span><span class="n">referer</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span> <span class="n">Available</span> <span class="n">Scrapy</span> <span class="n">objects</span><span class="p">:</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">scrapy</span>     <span class="n">scrapy</span> <span class="n">module</span> <span class="p">(</span><span class="n">contains</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">,</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Selector</span><span class="p">,</span> <span class="n">etc</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">crawler</span>    <span class="o">&lt;</span><span class="n">scrapy</span><span class="o">.</span><span class="n">crawler</span><span class="o">.</span><span class="n">Crawler</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7fa91d888c90</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">item</span>       <span class="p">{}</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">request</span>    <span class="o">&lt;</span><span class="n">GET</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">1</span><span class="o">/&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">response</span>   <span class="o">&lt;</span><span class="mi">200</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">1</span><span class="o">/&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">settings</span>   <span class="o">&lt;</span><span class="n">scrapy</span><span class="o">.</span><span class="n">settings</span><span class="o">.</span><span class="n">Settings</span> <span class="nb">object</span> <span class="n">at</span> <span class="mh">0x7fa91d888c10</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">spider</span>     <span class="o">&lt;</span><span class="n">DefaultSpider</span> <span class="s1">&#39;default&#39;</span> <span class="n">at</span> <span class="mh">0x7fa91c8af990</span><span class="o">&gt;</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span> <span class="n">Useful</span> <span class="n">shortcuts</span><span class="p">:</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">shelp</span><span class="p">()</span>           <span class="n">Shell</span> <span class="n">help</span> <span class="p">(</span><span class="nb">print</span> <span class="n">this</span> <span class="n">help</span><span class="p">)</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">fetch</span><span class="p">(</span><span class="n">req_or_url</span><span class="p">)</span> <span class="n">Fetch</span> <span class="n">request</span> <span class="p">(</span><span class="ow">or</span> <span class="n">URL</span><span class="p">)</span> <span class="ow">and</span> <span class="n">update</span> <span class="n">local</span> <span class="n">objects</span>
<span class="p">[</span><span class="n">s</span><span class="p">]</span>   <span class="n">view</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>    <span class="n">View</span> <span class="n">response</span> <span class="ow">in</span> <span class="n">a</span> <span class="n">browser</span>
</pre></div>
</div>
<p>使用shell，可以尝试使用 <a class="reference external" href="https://www.w3.org/TR/selectors">CSS</a> 对于响应对象：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title&#39;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&#39;descendant-or-self::title&#39; data=&#39;&lt;title&gt;Quotes to Scrape&lt;/title&gt;&#39;&gt;]</span>
</pre></div>
</div>
<p>运行``response.css（'title'）``的结果是一个类似于列表的对象：class：<cite>~scrapy.selector.SelectorList</cite>，它表示一个列表：class：<a href="#id1"><span class="problematic" id="id2">`</span></a>~scrapy.selector.Selector，这些对象环绕XML/HTML元素，并允许您运行进一步的查询，以细化所选内容或提取数据。</p>
<p>要从上述标题中提取文本，可以执行以下操作：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;Quotes to Scrape&#39;]</span>
</pre></div>
</div>
<p>这里有两件事需要注意：一是我们已经添加了 <code class="docutils literal notranslate"><span class="pre">::text</span></code> 对于CSS查询，意味着我们只想直接选择内部的文本元素 <code class="docutils literal notranslate"><span class="pre">&lt;title&gt;</span></code> 元素。如果我们不指定 <code class="docutils literal notranslate"><span class="pre">::text</span></code> ，我们将获得完整的title元素，包括其标记：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="go">[&#39;&lt;title&gt;Quotes to Scrape&lt;/title&gt;&#39;]</span>
</pre></div>
</div>
<p>另一件事是呼叫的结果 <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 是一个列表：选择器可能返回多个结果，因此我们提取所有结果。当您知道您只想要第一个结果时，如本例所示，您可以：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Quotes to Scrape&#39;</span>
</pre></div>
</div>
<p>作为替代，你可以写下：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Quotes to Scrape&#39;</span>
</pre></div>
</div>
<p>然而，使用 <code class="docutils literal notranslate"><span class="pre">.get()</span></code> 直接在A上 <a class="reference internal" href="../topics/selectors.html#scrapy.selector.SelectorList" title="scrapy.selector.SelectorList"><code class="xref py py-class docutils literal notranslate"><span class="pre">SelectorList</span></code></a> 实例避免了 <code class="docutils literal notranslate"><span class="pre">IndexError</span></code> 回报 <code class="docutils literal notranslate"><span class="pre">None</span></code> 当它找不到任何与所选内容匹配的元素时。</p>
<p>这里有一个教训：对于大多数抓取代码，您希望它能够对由于在页面上找不到的东西而导致的错误具有弹性，这样即使某些部分无法抓取，您至少可以 <strong>some</strong> 数据。</p>
<p>除此之外 <a class="reference internal" href="../topics/selectors.html#scrapy.selector.SelectorList.getall" title="scrapy.selector.SelectorList.getall"><code class="xref py py-meth docutils literal notranslate"><span class="pre">getall()</span></code></a> 和 <a class="reference internal" href="../topics/selectors.html#scrapy.selector.SelectorList.get" title="scrapy.selector.SelectorList.get"><code class="xref py py-meth docutils literal notranslate"><span class="pre">get()</span></code></a> 方法，也可以使用 <a class="reference internal" href="../topics/selectors.html#scrapy.selector.SelectorList.re" title="scrapy.selector.SelectorList.re"><code class="xref py py-meth docutils literal notranslate"><span class="pre">re()</span></code></a> 提取方法 <a class="reference external" href="https://docs.python.org/3/library/re.html" title="(在 Python v3.9)"><span class="xref std std-doc">regular expressions</span></a> ：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">re</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;Quotes.*&#39;</span><span class="p">)</span>
<span class="go">[&#39;Quotes to Scrape&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">re</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;Q\w+&#39;</span><span class="p">)</span>
<span class="go">[&#39;Quotes&#39;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;title::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">re</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;(\w+) to (\w+)&#39;</span><span class="p">)</span>
<span class="go">[&#39;Quotes&#39;, &#39;Scrape&#39;]</span>
</pre></div>
</div>
<p>为了找到合适的CSS选择器，您可能会发现在Web浏览器的shell中使用 <code class="docutils literal notranslate"><span class="pre">view(response)</span></code> . 您可以使用浏览器的开发人员工具检查HTML并找到一个选择器（请参见 <a class="reference internal" href="../topics/developer-tools.html#topics-developer-tools"><span class="std std-ref">使用浏览器的开发人员工具进行抓取</span></a> ）</p>
<p><a class="reference external" href="https://selectorgadget.com/">Selector Gadget</a> 也是一个很好的工具，可以快速找到视觉上选中的元素的CSS选择器，它可以在许多浏览器中使用。</p>
<div class="section" id="xpath-a-brief-intro">
<h4>XPath: 简介<a class="headerlink" href="#xpath-a-brief-intro" title="永久链接至标题">¶</a></h4>
<p>此外 <a class="reference external" href="https://www.w3.org/TR/selectors">CSS</a> ，scrapy选择器也支持使用 <a class="reference external" href="https://www.w3.org/TR/xpath/all/">XPath</a> 表达：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title&#39;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&#39;//title&#39; data=&#39;&lt;title&gt;Quotes to Scrape&lt;/title&gt;&#39;&gt;]</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">xpath</span><span class="p">(</span><span class="s1">&#39;//title/text()&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;Quotes to Scrape&#39;</span>
</pre></div>
</div>
<p>XPath表达式是非常强大的，是抓取选择器的基础。实际上，CSS选择器在引擎盖下转换为xpath。如果仔细阅读shell中选择器对象的文本表示形式，可以看到这一点。</p>
<p>虽然可能不像CSS选择器那么流行，但xpath表达式提供了更多的功能，因为除了导航结构之外，它还可以查看内容。使用xpath，您可以选择如下内容：<a href="#id1"><span class="problematic" id="id2">*</span></a>选择包含文本“下一页”<a href="#id3"><span class="problematic" id="id4">*</span></a>的链接。这使得xpath非常适合于抓取任务，并且我们鼓励您学习xpath，即使您已经知道如何构造css选择器，它也会使抓取更加容易。</p>
<p>我们在这里不会涉及很多XPath，但你可以阅读更多关于：ref：<cite>在这里使用带有Scrapy选择器的XPath &lt;topics-selectors&gt;</cite>。 要了解有关XPath的更多信息，我们建议`本教程通过示例学习XPath &lt;<a class="reference external" href="http://zvon.org/comp/r/tut-XPath_1.html">http://zvon.org/comp/r/tut-XPath_1.html</a>&gt;`_，以及`本教程学习“如何在XPath中思考 “&lt;<a class="reference external" href="http://plasmasturm.org/log/xpath101/">http://plasmasturm.org/log/xpath101/</a>&gt;`_。</p>
</div>
<div class="section" id="extracting-quotes-and-authors">
<h4>提取引用和作者<a class="headerlink" href="#extracting-quotes-and-authors" title="永久链接至标题">¶</a></h4>
<p>既然您对选择和提取有了一些了解，那么让我们通过编写代码从网页中提取引号来完成蜘蛛程序。</p>
<p><a class="reference external" href="Http://quotes.toscrape">Http://quotes.toscrape</a>.com中的每个引号都由如下所示的HTML元素表示：</p>
<div class="highlight-html notranslate"><div class="highlight"><pre><span></span><span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;quote&quot;</span><span class="p">&gt;</span>
    <span class="p">&lt;</span><span class="nt">span</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;text&quot;</span><span class="p">&gt;</span>“The world as we have created it is a process of our
    thinking. It cannot be changed without changing our thinking.”<span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;</span>
    <span class="p">&lt;</span><span class="nt">span</span><span class="p">&gt;</span>
        by <span class="p">&lt;</span><span class="nt">small</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;author&quot;</span><span class="p">&gt;</span>Albert Einstein<span class="p">&lt;/</span><span class="nt">small</span><span class="p">&gt;</span>
        <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&quot;/author/Albert-Einstein&quot;</span><span class="p">&gt;</span>(about)<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
    <span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;</span>
    <span class="p">&lt;</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;tags&quot;</span><span class="p">&gt;</span>
        Tags:
        <span class="p">&lt;</span><span class="nt">a</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;tag&quot;</span> <span class="na">href</span><span class="o">=</span><span class="s">&quot;/tag/change/page/1/&quot;</span><span class="p">&gt;</span>change<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
        <span class="p">&lt;</span><span class="nt">a</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;tag&quot;</span> <span class="na">href</span><span class="o">=</span><span class="s">&quot;/tag/deep-thoughts/page/1/&quot;</span><span class="p">&gt;</span>deep-thoughts<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
        <span class="p">&lt;</span><span class="nt">a</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;tag&quot;</span> <span class="na">href</span><span class="o">=</span><span class="s">&quot;/tag/thinking/page/1/&quot;</span><span class="p">&gt;</span>thinking<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
        <span class="p">&lt;</span><span class="nt">a</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;tag&quot;</span> <span class="na">href</span><span class="o">=</span><span class="s">&quot;/tag/world/page/1/&quot;</span><span class="p">&gt;</span>world<span class="p">&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
    <span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">div</span><span class="p">&gt;</span>
</pre></div>
</div>
<p>让我们打开Scrapy Shell并播放一点以了解如何提取所需数据：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ scrapy shell &#39;http://quotes.toscrape.com&#39;
</pre></div>
</div>
<p>我们得到了一个quote HTML元素的选择器列表，其中包括：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;div.quote&quot;</span><span class="p">)</span>
<span class="go">[&lt;Selector xpath=&quot;descendant-or-self::div[@class and contains(concat(&#39; &#39;, normalize-space(@class), &#39; &#39;), &#39; quote &#39;)]&quot; data=&#39;&lt;div class=&quot;quote&quot; itemscope itemtype...&#39;&gt;,</span>
<span class="go"> &lt;Selector xpath=&quot;descendant-or-self::div[@class and contains(concat(&#39; &#39;, normalize-space(@class), &#39; &#39;), &#39; quote &#39;)]&quot; data=&#39;&lt;div class=&quot;quote&quot; itemscope itemtype...&#39;&gt;,</span>
<span class="go"> ...]</span>
</pre></div>
</div>
<p>上面查询返回的每个选择器都允许我们对其子元素运行进一步的查询。让我们将第一个选择器分配给一个变量，这样我们就可以直接在特定的引号上运行CSS选择器：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">quote</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;div.quote&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
</pre></div>
</div>
<p>现在，让我们提取 <code class="docutils literal notranslate"><span class="pre">text</span></code> ， <code class="docutils literal notranslate"><span class="pre">author</span></code> 以及 <code class="docutils literal notranslate"><span class="pre">tags</span></code> 从引用中使用 <code class="docutils literal notranslate"><span class="pre">quote</span></code> 我们刚刚创建的对象：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">text</span> <span class="o">=</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;span.text::text&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">text</span>
<span class="go">&#39;“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”&#39;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">author</span> <span class="o">=</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;small.author::text&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">author</span>
<span class="go">&#39;Albert Einstein&#39;</span>
</pre></div>
</div>
<p>鉴于标记是字符串列表，我们可以使用 <code class="docutils literal notranslate"><span class="pre">.getall()</span></code> 方法获取所有这些参数：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">tags</span> <span class="o">=</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;div.tags a.tag::text&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">tags</span>
<span class="go">[&#39;change&#39;, &#39;deep-thoughts&#39;, &#39;thinking&#39;, &#39;world&#39;]</span>
</pre></div>
</div>
<p>找到了如何提取每个位之后，我们现在可以迭代所有的quotes元素，并将它们放在Python字典中：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;div.quote&quot;</span><span class="p">):</span>
<span class="gp">... </span>    <span class="n">text</span> <span class="o">=</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;span.text::text&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">... </span>    <span class="n">author</span> <span class="o">=</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;small.author::text&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="gp">... </span>    <span class="n">tags</span> <span class="o">=</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s2">&quot;div.tags a.tag::text&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">()</span>
<span class="gp">... </span>    <span class="nb">print</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="n">text</span><span class="p">,</span> <span class="n">author</span><span class="o">=</span><span class="n">author</span><span class="p">,</span> <span class="n">tags</span><span class="o">=</span><span class="n">tags</span><span class="p">))</span>
<span class="go">{&#39;text&#39;: &#39;“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”&#39;, &#39;author&#39;: &#39;Albert Einstein&#39;, &#39;tags&#39;: [&#39;change&#39;, &#39;deep-thoughts&#39;, &#39;thinking&#39;, &#39;world&#39;]}</span>
<span class="go">{&#39;text&#39;: &#39;“It is our choices, Harry, that show what we truly are, far more than our abilities.”&#39;, &#39;author&#39;: &#39;J.K. Rowling&#39;, &#39;tags&#39;: [&#39;abilities&#39;, &#39;choices&#39;]}</span>
<span class="gp">...</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="extracting-data-in-our-spider">
<h3>在蜘蛛中提取数据<a class="headerlink" href="#extracting-data-in-our-spider" title="永久链接至标题">¶</a></h3>
<p>让我们回到蜘蛛身边。到目前为止，它还没有提取任何数据，特别是将整个HTML页面保存到一个本地文件中。让我们把上面的提取逻辑集成到蜘蛛中。</p>
<p>剪贴蜘蛛通常会生成许多字典，其中包含从页面中提取的数据。为此，我们使用 <code class="docutils literal notranslate"><span class="pre">yield</span></code> 回调中的python关键字，如下所示：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">QuotesSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;quotes&quot;</span>
    <span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s1">&#39;http://quotes.toscrape.com/page/1/&#39;</span><span class="p">,</span>
        <span class="s1">&#39;http://quotes.toscrape.com/page/2/&#39;</span><span class="p">,</span>
    <span class="p">]</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.quote&#39;</span><span class="p">):</span>
            <span class="k">yield</span> <span class="p">{</span>
                <span class="s1">&#39;text&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;span.text::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;author&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;small.author::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.tags a.tag::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">(),</span>
            <span class="p">}</span>
</pre></div>
</div>
<p>如果运行这个spider，它将用日志输出提取的数据：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">2016</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">19</span> <span class="mi">18</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mi">19</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">scraper</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Scraped</span> <span class="kn">from</span> <span class="o">&lt;</span><span class="mi">200</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">1</span><span class="o">/&gt;</span>
<span class="p">{</span><span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;life&#39;</span><span class="p">,</span> <span class="s1">&#39;love&#39;</span><span class="p">],</span> <span class="s1">&#39;author&#39;</span><span class="p">:</span> <span class="s1">&#39;André Gide&#39;</span><span class="p">,</span> <span class="s1">&#39;text&#39;</span><span class="p">:</span> <span class="s1">&#39;“It is better to be hated for what you are than to be loved for what you are not.”&#39;</span><span class="p">}</span>
<span class="mi">2016</span><span class="o">-</span><span class="mi">09</span><span class="o">-</span><span class="mi">19</span> <span class="mi">18</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mi">19</span> <span class="p">[</span><span class="n">scrapy</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">scraper</span><span class="p">]</span> <span class="n">DEBUG</span><span class="p">:</span> <span class="n">Scraped</span> <span class="kn">from</span> <span class="o">&lt;</span><span class="mi">200</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">quotes</span><span class="o">.</span><span class="n">toscrape</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">page</span><span class="o">/</span><span class="mi">1</span><span class="o">/&gt;</span>
<span class="p">{</span><span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;edison&#39;</span><span class="p">,</span> <span class="s1">&#39;failure&#39;</span><span class="p">,</span> <span class="s1">&#39;inspirational&#39;</span><span class="p">,</span> <span class="s1">&#39;paraphrased&#39;</span><span class="p">],</span> <span class="s1">&#39;author&#39;</span><span class="p">:</span> <span class="s1">&#39;Thomas A. Edison&#39;</span><span class="p">,</span> <span class="s1">&#39;text&#39;</span><span class="p">:</span> <span class="s2">&quot;“I have not failed. I&#39;ve just found 10,000 ways that won&#39;t work.”&quot;</span><span class="p">}</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="storing-the-scraped-data">
<span id="storing-data"></span><h2>存储抓取的数据<a class="headerlink" href="#storing-the-scraped-data" title="永久链接至标题">¶</a></h2>
<p>存储抓取数据的最简单方法是使用 <a class="reference internal" href="../topics/feed-exports.html#topics-feed-exports"><span class="std std-ref">Feed exports</span></a> ，使用以下命令：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">crawl</span> <span class="n">quotes</span> <span class="o">-</span><span class="n">O</span> <span class="n">quotes</span><span class="o">.</span><span class="n">json</span>
</pre></div>
</div>
<p>这将生成一个``quotes.json``文件，其中包含所有已删除的项目，在`JSON`_中序列化。</p>
<p>这个 <code class="docutils literal notranslate"><span class="pre">-O</span></code> 命令行开关覆盖任何现有文件；使用 <code class="docutils literal notranslate"><span class="pre">-o</span></code> 而是将新内容附加到任何现有文件中。但是，附加到JSON文件会使文件内容无效JSON。附加到文件时，请考虑使用不同的序列化格式，例如 <a class="reference external" href="http://jsonlines.org">JSON Lines</a> ：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">crawl</span> <span class="n">quotes</span> <span class="o">-</span><span class="n">o</span> <span class="n">quotes</span><span class="o">.</span><span class="n">jl</span>
</pre></div>
</div>
<p>这个 <a class="reference external" href="http://jsonlines.org">JSON Lines</a> 格式很有用，因为它类似于流，您可以很容易地向它附加新记录。当您运行两次时，它不存在相同的JSON问题。另外，由于每个记录都是单独的一行，因此您可以处理大文件，而不必将所有内容都放入内存中，因此有如下工具： <a class="reference external" href="https://stedolan.github.io/jq">JQ</a> 以帮助在命令行中执行此操作。</p>
<p>在小项目中（如本教程中的项目），这就足够了。但是，如果您想对爬取的项目执行更复杂的操作，可以编写一个 <a class="reference internal" href="../topics/item-pipeline.html#topics-item-pipeline"><span class="std std-ref">Item Pipeline</span></a> . 项目创建时已为您设置了项目管道的占位符文件，位于 <code class="docutils literal notranslate"><span class="pre">tutorial/pipelines.py</span></code> . 但是，如果只想存储爬取的项目，则不需要实现任何项目管道。</p>
</div>
<div class="section" id="following-links">
<h2>以下链接<a class="headerlink" href="#following-links" title="永久链接至标题">¶</a></h2>
<p>比如说，你不需要从http://quotes.toscrape.com的前两页抓取内容，而是需要从网站上所有页面的引用。</p>
<p>既然您知道了如何从页面中提取数据，那么让我们看看如何从页面中跟踪链接。</p>
<p>第一件事是提取到我们要跟踪的页面的链接。检查我们的页面，我们可以看到有一个链接指向下一个带有以下标记的页面：</p>
<div class="highlight-html notranslate"><div class="highlight"><pre><span></span><span class="p">&lt;</span><span class="nt">ul</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;pager&quot;</span><span class="p">&gt;</span>
    <span class="p">&lt;</span><span class="nt">li</span> <span class="na">class</span><span class="o">=</span><span class="s">&quot;next&quot;</span><span class="p">&gt;</span>
        <span class="p">&lt;</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">&quot;/page/2/&quot;</span><span class="p">&gt;</span>Next <span class="p">&lt;</span><span class="nt">span</span> <span class="na">aria-hidden</span><span class="o">=</span><span class="s">&quot;true&quot;</span><span class="p">&gt;</span><span class="ni">&amp;rarr;</span><span class="p">&lt;/</span><span class="nt">span</span><span class="p">&gt;&lt;/</span><span class="nt">a</span><span class="p">&gt;</span>
    <span class="p">&lt;/</span><span class="nt">li</span><span class="p">&gt;</span>
<span class="p">&lt;/</span><span class="nt">ul</span><span class="p">&gt;</span>
</pre></div>
</div>
<p>我们可以尝试在外壳中提取：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;&lt;a href=&quot;/page/2/&quot;&gt;Next &lt;span aria-hidden=&quot;true&quot;&gt;→&lt;/span&gt;&lt;/a&gt;&#39;</span>
</pre></div>
</div>
<p>这将获取anchor元素，但我们需要该属性 <code class="docutils literal notranslate"><span class="pre">href</span></code> . 为此，Scrapy支持CSS扩展，允许您选择属性内容，如下所示：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="go">&#39;/page/2/&#39;</span>
</pre></div>
</div>
<p>还有一个 <code class="docutils literal notranslate"><span class="pre">attrib</span></code> 可用属性（请参见 <a class="reference internal" href="../topics/selectors.html#selecting-attributes"><span class="std std-ref">选择元素属性</span></a> 更多信息）：</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">attrib</span><span class="p">[</span><span class="s1">&#39;href&#39;</span><span class="p">]</span>
<span class="go">&#39;/page/2/&#39;</span>
</pre></div>
</div>
<p>现在让我们看看我们的spider被修改为递归地跟踪下一页的链接，从中提取数据：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">QuotesSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;quotes&quot;</span>
    <span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s1">&#39;http://quotes.toscrape.com/page/1/&#39;</span><span class="p">,</span>
    <span class="p">]</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.quote&#39;</span><span class="p">):</span>
            <span class="k">yield</span> <span class="p">{</span>
                <span class="s1">&#39;text&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;span.text::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;author&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;small.author::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.tags a.tag::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">(),</span>
            <span class="p">}</span>

        <span class="n">next_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
        <span class="k">if</span> <span class="n">next_page</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
            <span class="n">next_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">urljoin</span><span class="p">(</span><span class="n">next_page</span><span class="p">)</span>
            <span class="k">yield</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">next_page</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
<p>现在，在提取数据之后， <code class="docutils literal notranslate"><span class="pre">parse()</span></code> 方法查找到下一页的链接，并使用 <a class="reference internal" href="../topics/request-response.html#scrapy.http.Response.urljoin" title="scrapy.http.Response.urljoin"><code class="xref py py-meth docutils literal notranslate"><span class="pre">urljoin()</span></code></a> 方法（因为链接可以是相对的），并生成对下一页的新请求，将自身注册为回调，以处理下一页的数据提取，并保持爬行在所有页中进行。</p>
<p>这里您看到的是scrapy的以下链接机制：当您在回调方法中生成一个请求时，scrapy将计划发送该请求，并注册一个回调方法，以便在该请求完成时执行。</p>
<p>使用它，您可以构建复杂的爬虫程序，这些爬虫程序根据您定义的规则跟踪链接，并根据所访问的页面提取不同类型的数据。</p>
<p>在我们的示例中，它创建了一种循环，跟踪到下一页的所有链接，直到找不到一个为止——这对于爬行博客、论坛和其他带有分页的站点很方便。</p>
<div class="section" id="a-shortcut-for-creating-requests">
<span id="response-follow-example"></span><h3>创建请求的快捷方式<a class="headerlink" href="#a-shortcut-for-creating-requests" title="永久链接至标题">¶</a></h3>
<p>作为创建请求对象的快捷方式，您可以使用 <a class="reference internal" href="../topics/request-response.html#scrapy.http.TextResponse.follow" title="scrapy.http.TextResponse.follow"><code class="xref py py-meth docutils literal notranslate"><span class="pre">response.follow</span></code></a> ：：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">QuotesSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;quotes&quot;</span>
    <span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span>
        <span class="s1">&#39;http://quotes.toscrape.com/page/1/&#39;</span><span class="p">,</span>
    <span class="p">]</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.quote&#39;</span><span class="p">):</span>
            <span class="k">yield</span> <span class="p">{</span>
                <span class="s1">&#39;text&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;span.text::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;author&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;span small::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;tags&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.tags a.tag::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">getall</span><span class="p">(),</span>
            <span class="p">}</span>

        <span class="n">next_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
        <span class="k">if</span> <span class="n">next_page</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
            <span class="k">yield</span> <span class="n">response</span><span class="o">.</span><span class="n">follow</span><span class="p">(</span><span class="n">next_page</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
<p>不像Scrapy.Request， <code class="docutils literal notranslate"><span class="pre">response.follow</span></code> 直接支持相对URL-无需调用URLJOIN。注意 <code class="docutils literal notranslate"><span class="pre">response.follow</span></code> 只返回一个请求实例；您仍然需要生成这个请求。</p>
<p>也可以将选择器传递给 <code class="docutils literal notranslate"><span class="pre">response.follow</span></code> 而不是字符串；此选择器应提取必要的属性：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">href</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;ul.pager a::attr(href)&#39;</span><span class="p">):</span>
    <span class="k">yield</span> <span class="n">response</span><span class="o">.</span><span class="n">follow</span><span class="p">(</span><span class="n">href</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
<p>为了 <code class="docutils literal notranslate"><span class="pre">&lt;a&gt;</span></code> 元素有一个快捷方式： <code class="docutils literal notranslate"><span class="pre">response.follow</span></code> 自动使用其href属性。因此代码可以进一步缩短：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;ul.pager a&#39;</span><span class="p">):</span>
    <span class="k">yield</span> <span class="n">response</span><span class="o">.</span><span class="n">follow</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
<p>要从iterable创建多个请求，可以使用 <a class="reference internal" href="../topics/request-response.html#scrapy.http.TextResponse.follow_all" title="scrapy.http.TextResponse.follow_all"><code class="xref py py-meth docutils literal notranslate"><span class="pre">response.follow_all</span></code></a> 取而代之的是：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">anchors</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;ul.pager a&#39;</span><span class="p">)</span>
<span class="k">yield from</span> <span class="n">response</span><span class="o">.</span><span class="n">follow_all</span><span class="p">(</span><span class="n">anchors</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
<p>或者，进一步缩短：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">yield from</span> <span class="n">response</span><span class="o">.</span><span class="n">follow_all</span><span class="p">(</span><span class="n">css</span><span class="o">=</span><span class="s1">&#39;ul.pager a&#39;</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="more-examples-and-patterns">
<h3>更多示例和模式<a class="headerlink" href="#more-examples-and-patterns" title="永久链接至标题">¶</a></h3>
<p>下面是另一个spider，它演示回调和以下链接，这次是为了抓取作者信息：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">AuthorSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s1">&#39;author&#39;</span>

    <span class="n">start_urls</span> <span class="o">=</span> <span class="p">[</span><span class="s1">&#39;http://quotes.toscrape.com/&#39;</span><span class="p">]</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="n">author_page_links</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;.author + a&#39;</span><span class="p">)</span>
        <span class="k">yield from</span> <span class="n">response</span><span class="o">.</span><span class="n">follow_all</span><span class="p">(</span><span class="n">author_page_links</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_author</span><span class="p">)</span>

        <span class="n">pagination_links</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a&#39;</span><span class="p">)</span>
        <span class="k">yield from</span> <span class="n">response</span><span class="o">.</span><span class="n">follow_all</span><span class="p">(</span><span class="n">pagination_links</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">parse_author</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="k">def</span> <span class="nf">extract_with_css</span><span class="p">(</span><span class="n">query</span><span class="p">):</span>
            <span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="n">query</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="s1">&#39;&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>

        <span class="k">yield</span> <span class="p">{</span>
            <span class="s1">&#39;name&#39;</span><span class="p">:</span> <span class="n">extract_with_css</span><span class="p">(</span><span class="s1">&#39;h3.author-title::text&#39;</span><span class="p">),</span>
            <span class="s1">&#39;birthdate&#39;</span><span class="p">:</span> <span class="n">extract_with_css</span><span class="p">(</span><span class="s1">&#39;.author-born-date::text&#39;</span><span class="p">),</span>
            <span class="s1">&#39;bio&#39;</span><span class="p">:</span> <span class="n">extract_with_css</span><span class="p">(</span><span class="s1">&#39;.author-description::text&#39;</span><span class="p">),</span>
        <span class="p">}</span>
</pre></div>
</div>
<p>这个蜘蛛将从主页开始，它将跟踪所有指向作者页面的链接，调用 <code class="docutils literal notranslate"><span class="pre">parse_author</span></code> 它们的回调，以及与 <code class="docutils literal notranslate"><span class="pre">parse</span></code> 像我们以前看到的那样回拨。</p>
<p>这里，我们把回电传递给 <a class="reference internal" href="../topics/request-response.html#scrapy.http.TextResponse.follow_all" title="scrapy.http.TextResponse.follow_all"><code class="xref py py-meth docutils literal notranslate"><span class="pre">response.follow_all</span></code></a> 作为使代码更短的位置参数；它也适用于 <a class="reference internal" href="../topics/request-response.html#scrapy.http.Request" title="scrapy.http.Request"><code class="xref py py-class docutils literal notranslate"><span class="pre">Request</span></code></a> .</p>
<p>这个 <code class="docutils literal notranslate"><span class="pre">parse_author</span></code> 回调定义了一个助手函数，用于从CSS查询中提取和清理数据，并用作者数据生成python dict。</p>
<p>这个蜘蛛展示的另一个有趣的事情是，即使同一作者引用了很多话，我们也不需要担心多次访问同一作者页面。默认情况下，scrappy过滤掉对已经访问过的URL的重复请求，避免了由于编程错误而太多地访问服务器的问题。这可以通过设置进行配置 <a class="reference internal" href="../topics/settings.html#std-setting-DUPEFILTER_CLASS"><code class="xref std std-setting docutils literal notranslate"><span class="pre">DUPEFILTER_CLASS</span></code></a> .</p>
<p>希望到目前为止，您已经很好地了解了如何使用scrappy跟踪链接和回调的机制。</p>
<p>作为另一个利用以下链接机制的蜘蛛示例，请查看 <a class="reference internal" href="../topics/spiders.html#scrapy.spiders.CrawlSpider" title="scrapy.spiders.CrawlSpider"><code class="xref py py-class docutils literal notranslate"><span class="pre">CrawlSpider</span></code></a> 类，该类用于实现一个小规则引擎，您可以使用该引擎在上面编写爬虫程序。</p>
<p>另外，一个常见的模式是使用：ref：<a href="#id1"><span class="problematic" id="id2">`</span></a>trick将其他数据传递给回调&lt;topics-request-response-ref-request-callback-arguments&gt;`来构建包含来自多个页面的数据的项目。</p>
</div>
</div>
<div class="section" id="using-spider-arguments">
<h2>使用蜘蛛参数<a class="headerlink" href="#using-spider-arguments" title="永久链接至标题">¶</a></h2>
<p>通过使用 <code class="docutils literal notranslate"><span class="pre">-a</span></code> 运行它们时的选项：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scrapy</span> <span class="n">crawl</span> <span class="n">quotes</span> <span class="o">-</span><span class="n">O</span> <span class="n">quotes</span><span class="o">-</span><span class="n">humor</span><span class="o">.</span><span class="n">json</span> <span class="o">-</span><span class="n">a</span> <span class="n">tag</span><span class="o">=</span><span class="n">humor</span>
</pre></div>
</div>
<p>这些论点被传给蜘蛛 <code class="docutils literal notranslate"><span class="pre">__init__</span></code> 方法并默认成为spider属性。</p>
<p>在本例中，为 <code class="docutils literal notranslate"><span class="pre">tag</span></code> 参数将通过 <code class="docutils literal notranslate"><span class="pre">self.tag</span></code> . 您可以使用它使您的蜘蛛只获取带有特定标记的引号，并基于以下参数构建URL:：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">scrapy</span>


<span class="k">class</span> <span class="nc">QuotesSpider</span><span class="p">(</span><span class="n">scrapy</span><span class="o">.</span><span class="n">Spider</span><span class="p">):</span>
    <span class="n">name</span> <span class="o">=</span> <span class="s2">&quot;quotes&quot;</span>

    <span class="k">def</span> <span class="nf">start_requests</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">url</span> <span class="o">=</span> <span class="s1">&#39;http://quotes.toscrape.com/&#39;</span>
        <span class="n">tag</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">&#39;tag&#39;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">tag</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
            <span class="n">url</span> <span class="o">=</span> <span class="n">url</span> <span class="o">+</span> <span class="s1">&#39;tag/&#39;</span> <span class="o">+</span> <span class="n">tag</span>
        <span class="k">yield</span> <span class="n">scrapy</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">response</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">quote</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;div.quote&#39;</span><span class="p">):</span>
            <span class="k">yield</span> <span class="p">{</span>
                <span class="s1">&#39;text&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;span.text::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
                <span class="s1">&#39;author&#39;</span><span class="p">:</span> <span class="n">quote</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;small.author::text&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(),</span>
            <span class="p">}</span>

        <span class="n">next_page</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">css</span><span class="p">(</span><span class="s1">&#39;li.next a::attr(href)&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
        <span class="k">if</span> <span class="n">next_page</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
            <span class="k">yield</span> <span class="n">response</span><span class="o">.</span><span class="n">follow</span><span class="p">(</span><span class="n">next_page</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse</span><span class="p">)</span>
</pre></div>
</div>
<p>如果你通过 <code class="docutils literal notranslate"><span class="pre">tag=humor</span></code> 对于这个蜘蛛，您会注意到它只访问来自 <code class="docutils literal notranslate"><span class="pre">humor</span></code> 标记，如 <code class="docutils literal notranslate"><span class="pre">http://quotes.toscrape.com/tag/humor</span></code> .</p>
<p>你可以：参考：<cite>在这里学习更多关于处理蜘蛛参数的信息&lt;spiderargs&gt;</cite>。</p>
</div>
<div class="section" id="next-steps">
<h2>下一步<a class="headerlink" href="#next-steps" title="永久链接至标题">¶</a></h2>
<p>本教程只介绍 Scrapy 的基础知识，但这里没有提到很多其他特性。检查：ref：<a href="#id1"><span class="problematic" id="id2">`</span></a>topics-whatelse`部分：ref：<a href="#id3"><span class="problematic" id="id4">`</span></a>intro-overview`一章，快速概述最重要的部分。</p>
<p>您可以继续阅读以下部分：ref：<a href="#id1"><span class="problematic" id="id2">`</span></a>section-basics`以了解有关命令行工具，蜘蛛，选择器以及本教程尚未涵盖的其他内容的更多信息，例如对已删除数据进行建模。 如果您更喜欢使用示例项目，请查看：ref：<a href="#id3"><span class="problematic" id="id4">`</span></a>intro-examples`部分。</p>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="examples.html" class="btn btn-neutral float-right" title="实例" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="install.html" class="btn btn-neutral float-left" title="安装指南" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        
        &copy; 版权所有 2008–2020, Scrapy developers
      <span class="lastupdated">
        最后更新于 10月 18, 2020.
      </span>

    </p>
  </div>
    
    
    
    Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a
    
    <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a>
    
    provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
  
 
<script type="text/javascript">
!function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","page","once","off","on"];analytics.factory=function(t){return function(){var e=Array.prototype.slice.call(arguments);e.unshift(t);analytics.push(e);return analytics}};for(var t=0;t<analytics.methods.length;t++){var e=analytics.methods[t];analytics[e]=analytics.factory(e)}analytics.load=function(t){var e=document.createElement("script");e.type="text/javascript";e.async=!0;e.src=("https:"===document.location.protocol?"https://":"http://")+"cdn.segment.com/analytics.js/v1/"+t+"/analytics.min.js";var n=document.getElementsByTagName("script")[0];n.parentNode.insertBefore(e,n)};analytics.SNIPPET_VERSION="3.1.0";
analytics.load("8UDQfnf3cyFSTsM4YANnW5sXmgZVILbA");
analytics.page();
}}();

analytics.ready(function () {
    ga('require', 'linker');
    ga('linker:autoLink', ['scrapinghub.com', 'crawlera.com']);
});
</script>


</body>
</html>